Teleoperation Experiments with a Utah/MIT Hand and a 

VPL DataGlove 


D. Clark, J. Demmel, J. Hong 
G. LafFerriere, L. Salkind, X. Tan 


Robotics Research Laboratory 
Courant Institute 
New York University 
New York, NY 10012 

Abstract 

We describe a teieoperation system capable of controlling a Utah/MIT Dextrous Hand using 
a VPL DataGlove as a master. Additionally the system is capable of running the dextrous hand 
in robotic (autonomous) mode as new programs are developed. We present the software and 
hardware architecture used and describe the experiments performed. We further analyze the 
communication and calibration issues involved and investigate applications to the analysis and 
development of automated dextrous manipulations. 


1 Introduction 

The research in Dextrous robotic hands has been fueled by two main factors: the interest in 
more versatile manipulators and the availability of truly dextrous end effectors like the Utah/MIT 
hand [5] and the Salisbury hand [7]. Research has ranged from low level control [13] to high 
level task planning or grasping issues [8,12]. However, because of the large number of degrees of 
freedom to be controlled implementations of complex manipulations tasks have been few. The great 
similarity between the human hand and the Utah/MIT hand suggested that a master-slave pairing 
should be possible if we could measure the human joint angles accurately and in real time. The 
VPL DataGlove provides just that capability. 

At the Robotics Research Laboratory of New York University we have set up a hardware and 
software architecture suitable for teleoperating the Utah/MIT Hand with a VPL DataGlove. The 
Utah/MIT Hand is attached to a PUMA 560, whose position is also teleoperated. Thus, as the 
human wearer of the DataGlove moves his hand and fingers, the Utah/MIT Hand and PUMA 
mimic him closely. This work includes a sophisticated calibration procedure for the DataGlove and 
a supervisory real time operating system which made communication between the different system 
components possible. The current implementation allowed us to perform the following tasks: 

• stacking blocks 

• picking up a milk carton and pouring milk into a cup 

• picking up the cup using the handle and pouring the milk into another cup 

• picking up a nut and screwing it into a bolt. 



Moreover the system also allows the operation of the robotic hand in an autonomous mode 
providing for a smooth transition between the different modes. This lets the operator transfer 
control to preprogrammed motions when appropriate. The system will be described in the following 

sections. 

2 Other related work 

Our system is purely positioned controlled and uses only the operator’s visual feedback. We have 
not analyzed other feedback needed by the operator to facilitate the manipulation task nor have 
we developed a force reflecting master [1,6,11514]* Our work is, however, unique in providing the 
operator with a nearly anthropomorphic end effector and a very natural interface. 

The main emphasis of our work is in dextrous manipulation. Most research in that field is still 
in the exploratory stages, including building computer systems suitable for real time control [4]. 
Much of our effort has gone into this as well, as a result of which our system is quite flexible and 
fast; it will be compared to other systems in detail elsewhere [2,10]. As for accomplishing actual 
manipulation tasks using the Utah/MIT Hand, to our knowledge[4] our work is among the most 
successful. 

3 Hardware 

The hardware components of the system consist of: a VPL DataGlove, a Utah/MIT Hand, a PUMA 
560, a SUN workstation and several single board 68020 based computers. The communications 
between the different parts of the system were either through serial lines or through Ethernet (see 
Figure 1). 

The VPL DataGlove (or simply Glove) consists of a cloth glove with fiber-optic cables attached 
to the back and palm and passing over the finger joints (see Figure 2). The fiber-optic cables in the 
Glove are treated at the sites where fingers flex so that light escapes when a finger is bent. More 
light is lost when the movement is greater. Phototransistors convert the light into an electrical 
signal which is then digitized. Coupled with a Polhemus sensor (which can be mounted on the 
back of the Glove), the position and orientation of the hand in space (6 degrees of freedom from 
the Polhemus sensor) as well as 14 finger joint angles (from the DataGlove) can be measured in 
real time (about 60 times a second). 

The Utah/MIT hand (henceforth simply Hand) is a 16 degree of freedom manipulator consisting 
of 4 anthropomorphic fingers and a palm (see Figure 2). The fingers are arranged so that one of 
them opposes the other three much as in the human hand. The “thumb” however, has exactly 
the same design as the other fingers and it must always oppose them, not allowing for a flat palm 
configuration. Each finger has four joints and each joint is operated by a pair of opposing tendons 
connected to pneumatic pistons. It has position sensors at each joint and tension sensors for each 
tendon, mounted on the wrist. This sensory information is used by the controller to servo the joint 
position and/or joint torque. The actuator package is separated from the hand by an articulated 
extension linkage (the remotizer) which allows the hand to be conveniently attached to a robot arm. 
Control signals for the pneumatic pistons come from an analog controller box which is connected 
to a VME bus via D/A and A/D boards. Several 68020 based Ironies boards on the bus allow 
implementation of a variety of digital control schemes. A SUN workstation, connected to the VME 
bus via an HVE bus to bus connector, is used for program development and compilation. Programs 
are then run on the SUN or (more commonly) downloaded for running on the Ironies boards. In 
our current setup both the program that maps the Glove angles to Hand angles and a monitoring 
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Figure 1: Architecture of Hand Teleoperation System 
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program run on the SUN while a joint level servo loop and a mode- switching program run on the 
Ironies, 



Figure 2: The VPL DataGlove and the Utah/MIT Dextrous Hand attached to a PUMA 560 

The Hand is mounted on the end of a PUMA 560 robot arm. To control the position and 
orientation of the Hand in real-time, a 68020 generates new robot setpoints every 28 msecs based 
on the current Polhemus sensor readings. These setpoints are then sent over a serial line to the 
PUMA VAL-II controller using the ALTER interface, which performs independent joint PID control 
at a 1 msec servo loop rate. 

4 Software 

The software components of the system include SAGE, a real time operating system; Condor, a 
runtime environment for controlling the Hand; and HIC, a servo loop scheduler for hierarchical 
controllers. 

A major component of the remote teleoperation system is the SAGE operating system, which 
interfaces the Glove, arm, and Hand. SAGE is designed specifically for real-time robotics super- 
visory control, and has multi-tasking, memory management, low-overhead synchronization, and 
network communications capabilities. 

Both HIC and Condor run on four 68020 boards mounted on a VME bus. Condor [9] provides 
the development system, the inter-processor communication, the user interface, the timer interface, 
and the debugging support for control of the Utah/MIT hand. HIC provides the process or event 
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scheduler and the Periodic Data Queues used for inter-process communication. HIC’s scheduler 
performs the same functions as the Minimal Operating System (MOS) in Condor (and in a similar 
manner) but has more flexibility. 

An essential part of this teleoperation capability was a very complex calibration procedure 
developed for the DataGlove which mapped the sensor data to XJtah/M!IT hand joint angles in a 
way that looked natural to the wearer of the Glove. 

4.1 Raw joint servo. 

To control the Hand from the Glove only the lowest level of control is required of the Hand’s 
controller. This is called the raw joint servo since it works entirely in the Hand’s native joint 
position and tendon strain units. Figures 3,4 show the raw joint servo schematically and the 
communication between the servo and the controlling Sun. The box at the bottom of the figure 



Figure 3: Communication with the Joint Servo (teleoperated mode). 


represents the joint servo. This is a HIC process running on one of the Ironies processors. The three 
phases of the servo, input — planning — output, are represented by the top four sub-boxes. Notice 
that there are in fact two separate planners for the two different modes of operation (autonomous 
and teleoperated). The mode switching done by the switch event (box on center right) is achieved 
by changing the pointer for the current planner. The switch event is now triggered by a SUN 
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resident monitoring program (but could in fact be triggered from anywhere in the system). Some 
position offsets are also computed by the switch event in order to make the transition smooth (and 
are placed in the control parameters buffer). 



Figure 4: Communication with the Joint Servo (autonomous mode). 


Above the servo are the five periodic data queues (pdq’s) used by the servo (however only 
four of them are in use at a given time since the two planners use different target pdq’s). The 
input procedure obtains values from the joint position and tendon strain sensors and produces 
position and torque values which are placed in the actual position/torque pdq. The second phase, 
the planner^ takes these actual values and the target values and control parameters from the target 
position/torque pdq and the control parameter pdq respectively. The planner uses a combination 
of force and position control to produce trajectories for the joint actuators. The planner places the 
trajectories in the trajectory pdq. The output routine picks up the trajectories and applies them to 
the Hand’s actuators. This loop is repeated 250 times per second. 

The top two boxes in the figure represent programs running on the Sun3/160 host computer. 
The monitor program watches the actual, target, and control parameter pdq’s and displays their 
values. To control the Hand with the Glove a second program on the Sun is run which supplies 
target joint positions based on the sensed position of the operator’s joints in the glove. 

As higher level servo loops are implemented, controlling, say, the fingertip positions, they will 
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generate the joint level targets in autonomous mode. While no such servo is running, the joint level 
servo uses the last targets specified. 

The pdq’s are the unifying data structures in HIC. Their use provides a disciplined way for 
numerous multirate hierarchical servo loops to communicate in a timely way. 


5 Calibration and workspace constraints 

The calibration problems involved in interfacing the Glove with the Hand are several. First, the 
kinematic structures of the Hand and the human hand are different. Second, the Glove can only 
measure some of the joint angles of the human hand. Third, the Glove sensors do not measure 
individual joints separately but rather there are strong nonlinear correlations among the sensors. 
The most complex part of the calibration procedure turned out to be the elimination of these 
correlations. 

The calibration procedure is described in detail in [3]; we outline it here. The calibration 
procedure is decomposed into three levels. In the first level the raw sensor values are converted 
into joint angles by a parameter identification procedure. After considerable experimentation it 
was found that the relationship between raw output from the Glove sensors and the joint angles 
could be modeled satisfactorily by using a function of the form 

a — ar -\- b -\- clog(T’) 

where a is the joint angle, r is the sensor reading and a, 6, and c are parameters to be identified. 

In the second level the correlations between the different calibrated angles were eliminated. This 
was done by first identifying those joints which were correlated and then eliminating the influence 
of one of the angles on another one by one. It is important to note that these correlations are not 
linear. 

Finally, in the third level, we used forward kinematics on the human hand and inverse kinematics 
on the Hand to map the Glove angles to suitable Hand angles which would put the Hand fingertips 
in the same relative positions as the Glove fingertips. 

Because of limitations in the workspace of the remotizer constraints had to be put on the 
motion of the joints of the PUMA. Finding the appropriate set of constraints required solving the 
inverse kinematics for the remotizer. Those were approximated using a simple open kinematic chain 
model. The precise kinematics are much harder to describe because of flexibility of the rods and 
the complex arrangement of joint and links. 

6 Experiments 

With the setup described above we were able to successfully teleoperate the Hand using the Glove 
to supply finger angles, and the Polhemus sensor to supply position and orientation for the PUMA 
560 carrying the Hand. The following experiments were performed. 

1. Stacking blocks. Three two-inch cubic blocks were picked up using two or three fingers and 
then stacked up. The whole task took a trained operator 1 minute and 45 seconds to perform. 

2. Picking up a milk carton and pouring milk into a cup. A regular half gallon milk carton was 
opened by pushing back the flaps of the (already unglued) spout. Then it was picked up by 
grasping it from the side. After tilting it enough the milk poured into the cup without spilling 
out. The task took 2 minutes and 20 seconds to perform, including time to move to carton 
to a position where it could be grasped more firmly. 
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3. Picking up the cup using the handle and pouring milk into another cup. Three fingers were 
moved through the handle while the thumh pushed against the side of the cup. (1 minute 
and 50 seconds) 

4. Picking up a nut and screwing it onto a bolt (see Figure 2). A nut is picked up by the thumb 
and two fingers. It is then started on the 3/4in bolt and screwed in using two fingers. When 
successful it took a trained operator under minutes to perform. However starting the nut 
on the thread was difficult and 3 out of four times either the nut was dropped or the grip 
became unstable and the experiment had to be restarted. 

These experiments were performed fully in teleoperated mode and only visual feedback was used. 
Preliminary results using both the autonomous mode and a transition to a purely force controlled 
mode for releasing a grip indicate that we may greatly improve the performance in experiments 
such as 1 and 3 above. A force controlled mode can be achieved by setting the position feedback 
gain (specified in the control parameters data structure) to zero using the monitoring program. 

7 Applications 

The most immediate application we foresee for our current teleoperation setup is as an aid in 
developing automatic control schemes for dextrous manipulation tasks. Teleoperation can be used 
to perform the parts of a complex task which have not yet been automated. For example, in stacking 
up blocks we may teleoperate the hand until a grasp is achieved and then let a preprogrammed 
routine maintain it while we perform other motions with the whole hand. Then, when more fine 
positioning needs to be done to properly align the blocks control can again be transferred to the 
operator. Later when a grasping routine is written the initial teleoperation may be reduced to 
a simple pregrasping position at which point the autonomous program takes over and the rest 
proceeds as before. In this way smaller parts of a more complicated task can be tested before the 
whole task is automated. Our current setup also easily allows a mixed teleoperation/ automated 
mode. An example could be found in writing with the Hand. The compliance of the pen on the 
paper could be preprogrammed while the shaping of the characters would be teleoperated. 


8 Conclusions 

We described a teleoperation system for the Utah/MIT Dextrous hand using a VPL DataGlove. 
While the system could be greatly improved in many areas it provides a new facility with which to 
perform fine manipulation experiments. The ability to switch back and forth between a teleoperated 
mode and an automated mode both allows the operator to rest while some already preprogrammed 
motions are performed and helps in the development of complex dextrous manipulations. The 
teleoperated mode is facilitated by the use of the natural interface provided by the DataGlove. 


Acknowledgments 

Work on this paper has been possible by Office of Naval Research Contract :j^N00014-87-K-0129, 
National Science Foundation CER Grant #DCR-8320085, Carnegie Mellon University subcontract 
#CMU-406349-55586 (NSF), NASA grant #NAG 2-493 and by grants from the Digital Equipment 
Corporation and the IBM Corporation. The second author is a Presidential Young Investigator. 


88 



References 

[1] Antal K. Bejczy and Zoltan F. Szakaly. A laboratory system for generalized bilateral and shared 
manual^ computer control of robot arms. Publication 88-5, Jet Propulsion Laboratory, April 
1988. Preliminary draft, presented at the Workshop on Dextrous Hands, IEEE, Philadelphia, 
April 1988. 

[2] Dayton Clark. Hierarchical Control System. Robotics Report 167, New York University, 
Courant Institute, Robotics Research Laboratory, 1988. 

[3] Jiaiwei Hong and Xiaonan Tan. Teleoperating the Utah/MIT Hand with a VPL DataGlove. 
LDataGlove Calibration. Robotics Report 168, New York University, Courant Institute, 
Robotics Research Laboratory, New York, 1988. 

[4] T. Iberall and S.T. Venkataraman, editors. Workshop on Dextrous Robot Hands., IEEE, 
Philadelphia, April 1988. 

[5] Steve C. Jacobsen, John E. Wood, D.F. Knutti, and Klaus B. Biggers. The Utah/MIT dextrous 
hand: work in progress. In M. Brady and R.P. Paul, editors. First International Conference 
on Robotics Research, pages 601-653, MIT Press, Cambridge, Massachusetts, 1984. 

[6] S. Lee, G. Bekey, and A.K. Bejczy. Computer control of space-borne teleoperators with sensory 
feedback. In Proc. IEEE Int. Conf. on Robotics and Automation, pages 205-214, St. Louis, 
MO, March 1985. 

[7] Matthew T. Mason and J. Kenneth Salisbury, Jr. Robots Hands and the Mechanics of Manip- 
ulation. MIT Press, Cambridge, Massachusetts, 1985. 

[8] B. Mishra, J.T. Schwartz, and M. Sharir. On the Existence and Synthesis of Multifinger 
Positive Grips. Robotics Report 89, Courant Institute, NYU, New York, November 1986. 

[9] Sundar Narasimhan, David M. Siegel, David Taylor, and Steven M. Drucker. The Condor 
Programmer’s Manual - Version II. Artificial Intelligence Laboratory, MIT, Cambridge, Mas- 
sachusetts, 1987. 

[10] Lou Salkind. The SAGE Operating System. Robotics Report, New York University, Courant 
Institute, Robotics Research Laboratory, 1988. 

[11] Lawrence W. Stark, Won S. Kim, and Frank Tendick. Cooperative control in telerobotics. 
In Proc. IEEE Robotics and Automation Conference, pages 593—597, Philadelphia, PA, April 
1988. 

[12] Rajko Tomovic, George E. Bekey, and Walter J. Karplus. A strategy for grasp synthesis 
with a multifingered hand. In Proceedings of the International Conference on Robotics and 
Automation, pages 83-89, IEEE, Raleigh, North Carolina, April 1987. 

[13] S.T. Venkataraman and Theodore E. Djaferis. Multivariable feedback control of the 
JPL/Stanford hand. In Proceedings of the International Conference on Robotics and Au- 
tomation, pages 77-82, 1987. 

[14] Jean Vertut and Philippe Coiffet. Teleoperation and Robotics: Applications and Technology. 
Volume 3B of Robot Technology, Prentice Hall, 1986. 


89 



